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Fixed  Observers 

Expect«l  Curve  of  Pd  versus  Range  for  Fti«ed  Observers 
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Analysis  Techniques 

•Fitting  Logistic  Curve 
•Using  Fisher  Exact  Test 


Fitting  Logistic  Curves 


•Based  on  the  logit  transformation, 
logistic  curves  are  defined  by 


"  1 +*„(*,") 

Where  Pd  is  probability  of  detection, 
R  is  range,  and  the  fitting  parameters 
are  bo  and  bj. 


Family  of  Logistic  Curves 
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Family  of  Logistic  Curves 
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Output  fromFitting  Curves 
•Fitted  curves 

•Significance  of  curve  parameters, 
probability  that  fitted  parameters  are 
better  than  null  parameters. 

•No  value  for  significance  of 
difference  between  vehicles. 


Fisher  Exact  Test 


Vehicle  A 

Vehicle  B 

Total 

Detect 

15 

18 

33 

No  Detect 

24 

15 

39 

Total 

39 

33 

72 

Pd 

38% 

55% 

46% 

Null  Hypothesis:  Vehicle  A  and 
Vehicle  B  have  the  same  Pd. 


For  95%  confidence,  reject  null 
hypothesis  if  p(Detects<15)  >5% 


Fisher  Exact  Test 


Use  Hypergeometric  Distribution: 
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Fisher  Exact  Test 


Vehicle  A 

Vehicle  B 

Total 

Detect 

15 

18 

33 

No  Detect 

24 

15 

39 

Total 

39 

33 

72 

Pd 

38% 

55% 

46% 

p{Detects  <  15)  =  p(15)*0.5  +  /?(14)  +  ...  +  /7(0) 
=  9.2%  >  5% 


Fisher  Exact  Test 


Vehicle  A 

Vehicle  B 

Total 

Detect 

15 

18 

33 

No  Defect 

24 

15 

39 

Total 

39 

33 

72 

Pd 

38% 

55% 

46% 

We  reject  the  null  hypothesis. 

We  cannot  be  sure  with  95Vo  confidence 
that  Vehicle  A  is  less  detectable  than 
Vehicle  B. 


Comparison  of  Techniques 

•Only  Fisher  Exact  Test  allows 
hypothesis  testing  with  the  calculation 
of  significance  of  difference. 

•  Fitting  logistic  curves  allows 
interpolation  between  data  points  and 
extrapolation  beyond  the  data. 
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Detectability  Specifications 


•Max  Range  for  P^](50%) 
•Max  Pj  at  Given  Range 


Detectability  Specifications 

•Max  Range  for  P(j(50%) 

•Requires  interpolation  to  calculate 
a  range  between  measured  points. 

•Max  Pjj  at  Given  Range 

•Direct  calculation  at  a  measured 
range. 


Recommendations 

•To  compare  two  vehicles, 
observation  tests  should  be  analyzed 
by  the  Fisher  Exact  Test. 

•For  a  directly  testable  detectability 
specification,  the  maximum  at  a 
given  range  should  be  required. 
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ABSTRACT 


For  vehicle  designers,  the  main  goal  of  experiments  on  the  observability  of 
combat  vehicles  is  a  comparison  of  the  probability  of  detection  for  two 
vehicles  as  a  function  of  range.  This  paper  addresses  the  statistical 
techniques  to  analyze  such  data  to  compare  the  probability  of  detection  of 
the  two  vehicles  as  a  function  of  range.  Two  techniques  are  compared:  1. 
Fitting  logistic  curves  to  the  data  of  the  vehicles  and  2.  Using  the  Fish^ 
Exact  Test  to  compare  the  probability  of  detection  of  the  two  vehicles  at 
each  range.  The  paper  discusses  the  issues  of  background  variability  and 
confidence  levels  for  hypothesis  testing.  Finally,  a  recommendation  is 
made  on  how  to  write  a  specification  for  the  detectability  of  a  vehicle. 


1.  The  Experimental  Situation 

Consider  Figure  1,  a  sketch  of  a  typical  field  test  of  vehicle  detectability.  Typically,  the 
experimenter  offers  observers  a  large  number  of  views  of  each  vehicle  at  several  ranges  and  in  a 
variety  of  backgrounds.  The  experimenter  records  the  numbers  of  true,  felse  and  missed 
detections  of  the  target  vehicle. 

Note  that  in  these  experiments  with  fixed  observers  range  cannot  be  varied  independently  of 
background  conditions.  Figure  2  shows  an  experimental  setup  in  which  range  can  be  varied 
independently  of  the  background.  This  setup,  however,  requires  moving  the  observers  as  the 
target  remains  fixed.  Although  such  a  setup  could  in  princ4>al  be  employed,  costs  will  generally 
dictate  tests  with  fixed  observers. 

Such  a  moveable  observer  test  should  generate  a  monotonic  curve  of  probability  of  detection  (Pd) 
versus  range.  Figure  3,  because  range  has  varied  independently  of  the  vehicle  background.  On  the 
other  hand,  a  fixed  observer  test  will  generate  a  Pd  versus  range  curve  with  background 
variability  superimposed  on  a  monotonic  curve.  Figure  4. 


2.  Analysis  Techniques 


For  vehicle  designers,  the  main  goal  of  experiments  on  the  obswvability  of  combat  vehicles  is  a 
comparison  of  the  probability  of  detection  for  two  vehicles.  In  this  paper,  I  will  compare  two 
techniques  for  anafyzing  data  from  such  fixed  observer  tests: 

1 .  Fitting  logistic  curves  to  the  data  of  the  vehicles. 

2.  Using  the  Fisher  Exact  Test  to  compare  the  probability  of  detection  of  the  two 
vehicles  at  each  range. 

Logistic  Curves 

The  logistic  curve  arose  from  the  transformation  of  probabilities,  p,  by  the  so-called  logit 
transformation,  namely, 

T  =  log,-^ 

\-p 

The  logit  transformation  tends  to  linearize  data  in  which  a  proportion  is  a  function  of  another 
variable,  such  as  range. 

The  following  equation  defines  the  logistic  curve  itself: 


l  +  6o(6f) 

General  statistics  software  packages,  such  as  SPSS,  can  fit  a  logistic  curve  to  data  and  can 
calculate  a  measure  of  goodness  of  fit. 

Fisher  Exact  Test 

The  Fisher  Exact  Test  is  designed  to  compare  two  proportions  to  decide  if  the  proportions  are 
significantly  different.  The  test  begins  with  the  preparation  of  a  contingency  table.  For  observer 
tests,  a  separate  table  would  be  prepared  for  each  range. 

For  example,  consider  the  following  table; 


Vehicle  A 

Vehicle  B 

Total 

Detect 

a 

b 

a+b 

No  Detect 

c 

d 

c+d 

Total 

e 

f 

e+f 

Pd 

a/e 

b/f 

(a+bWe+f) 

The  Fisher  Exact  Test  calculates  the  probability  that  the  observed  contingency  table  or  a  more 
extreme  table  would  be  the  result  of  random  variations  of  draws  from  a  population  with  a  Pd  of 
(a+b)/(e+f).  The  Fisher  Exact  Test  uses  the  hypergeometric  distribution  to  calculate  the  required 
probabilities.  The  hypergeometric  distribution  describes  the  probability,  for  example,  of  drawing 
5  red  cards  in  a  1 3-card  hand  from  deck  of  26  red  and  26  black  cards. 


The  Fisher  Exact  Test  supports  direct  hypothesis  testing.  A  typical  null  hypothesis  would  be  that 
Vehicle  A  and  Vehicle  B  have  the  same  Pd.  The  experimenter  would  select  a  significance  level 
before  the  test,  for  example  5%.  If  the  observed  contingency  table  has  a  probability  less  than  5%, 
then  the  experimentw  can  state  with  greater  than  95%  confidence  that  the  two  vehicles  have 
different  Pd 's. 


3.  Discussion 

The  Fisher  Exact  Test  has  advantages  over  fitting  a  logistic  curve.  Because  data  is  compared  only 
at  the  same  range,  the  effect  variability  of  background  with  range  is  avoided.  Experimental 
sample  sizes  can  be  calculated  for  comparison  of  proportions  in  order  to  insure  a  given  difference 
in  Pd  will  be  significant.  Moreover,  the  Fisher  Exact  Test  lends  itself  to  quantitative  hypothesis 
testing. 


4.  Recommendation  on  Writing  Specifications 

The  analysis  of  observer  data  has  a  bearing  on  how  specifications  should  be  written.  To  use 
quantitative  hypothesis  tests  to  judge  compliance  to  specifications,  the  maximum  allowed  Pd  at 
given  range  should  be  specified  instead  of  the  maximum  allowed  range  at  Pd  of  50%.  Consider 
the  following  specifications:  1 .  “The  range  for  a  Pd  of  50%  must  be  less  than  R.”  and  2.  “The  Pd 
at  a  range  of  R  must  be  less  than  50%.”  The  two  specifications  are  equally  stringent,  but  the 
second  specification  allows  a  direct  test  with  a  quantitative  significance  level.  On  the  other  hand, 
the  only  way  to  determine  the  range  for  Pd  of  50%  is  through  interpolation  by  curve  fitting. 


Test  with  Fixed  Observers 


Fixed  Observers 


Figure  1 .  Sketch  of  an  obsCTver  test  using  fixed  observers. 


Figure  2.  Sketch  of  an  observer  test  with  a  fixed  target  vehicle. 


Fixed  Vehicle 


EvfwcM  Curv*  of  Pd  nnum  Range  for  Rxed  Vettkte 


Figure  3.  Curve  to  be  expected  from  a  test  using  a  fixed  target  vehicle. 


Fixed  Observers 
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Figure  4.  Curve  to  be  expected  from  an  observer  test  using  fixed  observo^. 


